LIVY-979: Create a helm-based integration environment for debugging Livy on Kubernetes - #537
LIVY-979: Create a helm-based integration environment for debugging Livy on Kubernetes#537nileshrathi345 wants to merge 1 commit into
Conversation
1b8bcf5 to
7d424f1
Compare
Add dev/helmchart/ to deploy Livy, Spark History Server, RBAC, ingress, and optional cert-manager on a local Kubernetes cluster (e.g. Docker Desktop). Includes ASF license headers, validation script, README/Docker.md, and review fixes: no vendored chart tgz, consistent resource naming, unified RBAC/TLS config, and derived image tags from sparkVersion. Builds on the original contribution in apache#454. Co-authored-by: Asif Khatri <123077165+askhatri@users.noreply.github.com>
7d424f1 to
7399865
Compare
| docs/**/*.html | ||
| docs/**/JB/** | ||
| venv/* | ||
| dev/helmchart/Chart.lock |
There was a problem hiding this comment.
License headers have been added to all files except this one, but since this file is generated by helm dependency build, any license headers are overwritten. I checked other Apache projects and adding this file into .rat-excludes seems to be common practice, so this matches the setup found in other projects:
https://github.com/apache/airflow/blob/8c16340ca4903e3974b5342c0fe8ca1f0534395f/.rat-excludes#L118
https://github.com/apache/pulsar-helm-chart/blob/328846044c8e1df50399c28bc63887696b6d58dc/.rat-excludes#L20
|
@gyogal, @roczei and @ArnavBalyan could please take a look when possible and review. |
Hi @nileshrathi345, I don't have experience with Helm charts, so I won't be able to review this one. |
3 participants
What changes were proposed in this pull request?
Adds a Helm chart under dev/helmchart/ for running Livy on a local Kubernetes cluster (e.g. Docker Desktop) for development and debugging, without cloud dependencies.
Included:
values.yamlaligned with Livy 1.0.0-SNAPSHOT, Spark 3.5.6, Scala 2.12README.md(install, smoke tests, optional Grafana/Loki),Docker.md(image build steps)test/validate.sh— ASF license header checks, version consistency, helm lint / helm template.rat-excludesupdated only for generatedChart.lockTemplate fixes found during local testing:
apiVersionrendering (trim directive)HADOOP_USER_NAME=rootfor local Hadoop user resolution)This PR is dev/ops tooling only — no changes to Livy server runtime code.
JIRA: LIVY-979
How was this patch tested?
Automated / local chart checks:
dev/helmchart/test/validate.sh— license headers, version keys, helm dependency build, helm lint, helm templateManual testing (Docker Desktop Kubernetes, namespace livy-dev)
Docker.md(livy-local/spark-py:v3.5.6,livy-local/livy:spark3.5.6)helm install/helm upgradelivycluster-0, ingress-nginx, cert-manager,spark-history-0GET /livy/version→ 2001+1,sc.parallelize(...).count(),spark.range(5).collect()— all succeededWas this patch authored or co-authored using generative AI tooling?
Yes, co-authored by Cursor AI
Note: This PR builds on the original Helm chart work contributed by @askhatri